Use fsid for path equality instead of storage_options#537
Open
kevinjacobs-delfi wants to merge 1 commit intofsspec:mainfrom
Open
Use fsid for path equality instead of storage_options#537kevinjacobs-delfi wants to merge 1 commit intofsspec:mainfrom
kevinjacobs-delfi wants to merge 1 commit intofsspec:mainfrom
Conversation
Change __eq__, relative_to, and is_relative_to to compare paths based on
filesystem identity (fsid) rather than storage_options directly.
This fixes the issue where paths to the same resource were considered
unequal due to differing non-identity options (auth, performance, etc.):
# Previously unequal, now equal (same S3 filesystem)
UPath('s3://bucket/file.txt') == UPath('s3://bucket/file.txt', anon=True)
The fsid is computed from protocol, storage_options, and fsspec global
config without instantiating the filesystem. For filesystems where fsid
cannot be determined, falls back to storage_options comparison.
Key changes:
- Add upath/_fsid.py with _fallback_fsid() for computing fsid
- Add fsid property to _UPathMixin and ProxyUPath
- Update __eq__ in UPath and LocalPath to use fsid
- Update relative_to and is_relative_to to use fsid
- Add tests for fsid-based equality
- Document behavior in migration guide and concepts docs
Closes fsspec#532
Co-Authored-By: Claude Opus 4.5 <[email protected]>
Collaborator
|
Thank you for the PR @kevinjacobs-delfi 🙏 ❤️ I'll get to reviewing it asap. It's a bit busy these days. |
Author
|
I can remove the use of the |
Collaborator
|
no need. The next release target will be And thanks for your patience. I might find some time this week to do a thorough review. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR changes
__eq__,relative_to, andis_relative_toto compare paths based on filesystem identity (fsid) rather thanstorage_optionsdirectly.Fixes #532
The Problem
Previously, paths to the same resource were considered unequal if their
storage_optionsdiffered:The Solution
Use fsid (filesystem identifier) to determine if two paths are on the same filesystem. The fsid ignores options that don't affect which filesystem is accessed (auth, performance settings) while considering options that do (endpoint_url, account_name, host+port).
Key Implementation Details
fsspec.config.conf) without instantiating the filesystemstorage_optionscomparisonfs.fsidwhich raisesNotImplementedError,UPath.fsidreturnsNoneLocalFileSystemandHTTPFileSystem_fallback_fsiddue to complexity (would need to handle mutable storage_options dicts) and the minimal cost of computing fsid on the fly. If needed,UPath.fsidcan be changed to a cached property in the future.Changes
upath/_fsid.pywith_fallback_fsid()for computing fsid from protocol + storage_options + global configfsidproperty to_UPathMixinandProxyUPath__eq__inUPathandLocalPathto use fsidrelative_toandis_relative_toto use fsidupath/tests/test_fsid.pyincluding audit testsdocs/migration.md) and concepts (docs/concepts/upath.md)Test plan
relative_toandis_relative_towith matching/different fsids🤖 Generated with Claude Code